How Similar Is Too Similar?: Exploring Users' Perceptions of Similarity in Playlist Evaluation

The Audio Music Similarity and Retrieval (AMS) task in the annual Music Information Retrieval eXchange relies on human-evaluation. One limitation of the current design of AMS is that evaluators are provided with scarce contextual information as to why they are evaluating the similarity of the songs and how this information will be used. This study explores the potential use of AMS results for generating playlists based on similarity. We asked participants to listen to a subset of results from the 2010 AMS task and evaluate the set of candidates generated by the algorithms as a playlist generated from a seed song (the query). We found that while similarity does affect how people feel about the candidate set as a playlist, other factors such as variety, metadata, personal preference, familiarity, mix of familiar and new music, etc. also strongly affect users' perceptions of playlist quality as well. We discuss six user behaviors in detail and the implications for the AMS evaluation task.